{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "raw"
    }
   },
   "source": [
    "# HW 2: Recipe Bot Error Analysis\n",
    "\n",
    "## 🎯 Assignment Overview\n",
    "\n",
    "This notebook helps you perform error analysis for your Recipe Bot by:\n",
    "\n",
    "1. **Part 1: Generate Test Queries** - Create diverse queries using key dimensions\n",
    "2. **Part 2: Run & Annotate** - Test your bot and identify failure patterns  \n",
    "3. **Part 3: Create Taxonomy** - Build structured failure mode categories\n",
    "\n",
    "**Goal:** Systematically identify what goes wrong with your bot and why.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/sallyanndelucia/miniconda3/envs/recipe-bot-hw2/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import required libraries\n",
    "import os\n",
    "import random\n",
    "import warnings\n",
    "from pathlib import Path\n",
    "\n",
    "import openai\n",
    "import pandas as pd\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "from phoenix.client import AsyncClient\n",
    "from phoenix.evals import OpenAIModel, PromptTemplate, llm_generate\n",
    "\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "# Load environment variables\n",
    "load_dotenv()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import getpass\n",
    "\n",
    "if \"OPENAI_API_KEY\" not in os.environ or not os.environ[\"OPENAI_API_KEY\"]:\n",
    "    os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"Enter your OPENAI_API_KEY: \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ Setup complete - Ready for error analysis with Phoenix!\n"
     ]
    }
   ],
   "source": [
    "# Configuration\n",
    "MODEL_NAME = \"gpt-4o-mini\"\n",
    "OUTPUT_DIR = Path(\"./data\")\n",
    "OUTPUT_DIR.mkdir(exist_ok=True)\n",
    "\n",
    "# Set up Phoenix OpenAI model\n",
    "phoenix_model = OpenAIModel(model=MODEL_NAME, temperature=0.9)\n",
    "\n",
    "print(\"✅ Setup complete - Ready for error analysis with Phoenix!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "raw"
    }
   },
   "source": [
    "# Part 1: Define Dimensions & Generate Initial Queries\n",
    "\n",
    "## Step 1.1: Identify Key Dimensions\n",
    "\n",
    "Identify 3-4 key dimensions relevant to your Recipe Bot's functionality. For each dimension, list at least 3 example values.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🎯 Defined key dimensions for Recipe Bot testing:\n",
      "   dietary_restriction: vegan, vegetarian, gluten-free, keto, no restrictions\n",
      "   cuisine_type: Italian, Asian, Mexican, Mediterranean, American, any cuisine\n",
      "   meal_type: breakfast, lunch, dinner, snack, dessert\n",
      "   skill_level: beginner, intermediate, advanced\n",
      "\n",
      "Total possible combinations: 450\n"
     ]
    }
   ],
   "source": [
    "# Define 4 key dimensions for Recipe Bot testing with specific values\n",
    "\n",
    "DIMENSIONS = {\n",
    "    \"dietary_restriction\": [\"vegan\", \"vegetarian\", \"gluten-free\", \"keto\", \"no restrictions\"],\n",
    "    \"cuisine_type\": [\"Italian\", \"Asian\", \"Mexican\", \"Mediterranean\", \"American\", \"any cuisine\"],\n",
    "    \"meal_type\": [\"breakfast\", \"lunch\", \"dinner\", \"snack\", \"dessert\"],\n",
    "    \"skill_level\": [\"beginner\", \"intermediate\", \"advanced\"],\n",
    "}\n",
    "\n",
    "print(\"🎯 Defined key dimensions for Recipe Bot testing:\")\n",
    "for dim, values in DIMENSIONS.items():\n",
    "    print(f\"   {dim}: {', '.join(values)}\")\n",
    "\n",
    "print(\n",
    "    f\"\\nTotal possible combinations: {len(DIMENSIONS['dietary_restriction']) * len(DIMENSIONS['cuisine_type']) * len(DIMENSIONS['meal_type']) * len(DIMENSIONS['skill_level'])}\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "raw"
    }
   },
   "source": [
    "## Step 1.2: Generate Unique Combinations (Tuples)\n",
    "\n",
    "Generate 15-20 unique combinations of these dimension values using programmatic sampling.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🎯 Generating 25 diverse dimension tuples programmatically...\n",
      "✅ Generated 25 diverse dimension tuples\n",
      "\n",
      "📋 Sample dimension tuples:\n",
      "\n",
      "Tuple 1: vegan, Italian, dinner, beginner\n",
      "\n",
      "Tuple 2: vegetarian, Asian, breakfast, advanced\n",
      "\n",
      "Tuple 3: no restrictions, Italian, dessert, intermediate\n",
      "\n",
      "Tuple 4: vegan, Italian, breakfast, beginner\n",
      "\n",
      "Tuple 5: vegetarian, American, dessert, beginner\n",
      "\n",
      "✅ Successfully created 25 diverse dimension tuples\n"
     ]
    }
   ],
   "source": [
    "# Step 1: Generate diverse dimension tuples programmatically to ensure variety\n",
    "print(\"🎯 Generating 25 diverse dimension tuples programmatically...\")\n",
    "\n",
    "# Create diverse combinations by sampling systematically\n",
    "dimension_tuples = []\n",
    "random.seed(42)  # For reproducible results\n",
    "\n",
    "# Generate 25 diverse tuples\n",
    "for i in range(25):\n",
    "    tuple_data = {\n",
    "        \"dietary_restriction\": random.choice(DIMENSIONS[\"dietary_restriction\"]),\n",
    "        \"cuisine_type\": random.choice(DIMENSIONS[\"cuisine_type\"]),\n",
    "        \"meal_type\": random.choice(DIMENSIONS[\"meal_type\"]),\n",
    "        \"skill_level\": random.choice(DIMENSIONS[\"skill_level\"]),\n",
    "        \"tuple_id\": i + 1,\n",
    "    }\n",
    "    dimension_tuples.append(tuple_data)\n",
    "\n",
    "print(f\"✅ Generated {len(dimension_tuples)} diverse dimension tuples\")\n",
    "\n",
    "# Step 2: Show some examples to verify diversity\n",
    "print(\"\\n📋 Sample dimension tuples:\")\n",
    "for i in range(min(5, len(dimension_tuples))):\n",
    "    tuple_data = dimension_tuples[i]\n",
    "    print(\n",
    "        f\"\\nTuple {i + 1}: {tuple_data['dietary_restriction']}, {tuple_data['cuisine_type']}, {tuple_data['meal_type']}, {tuple_data['skill_level']}\"\n",
    "    )\n",
    "\n",
    "print(f\"\\n✅ Successfully created {len(dimension_tuples)} diverse dimension tuples\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "raw"
    }
   },
   "source": [
    "## Step 1.3: Generate Natural Language User Queries\n",
    "\n",
    "Take 5-7 of the generated tuples and create a natural language user query for your Recipe Bot for each selected tuple. Review these generated queries to ensure they are realistic and representative of how a user might interact with your bot.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "🐌!! If running inside a notebook, patching the event loop with nest_asyncio will allow asynchronous eval submission, and is significantly faster. To patch the event loop, run `nest_asyncio.apply()`.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "📝 Selected 25 dimension tuples for query generation\n",
      "🎯 Converting dimension tuples to natural language queries...\n",
      "The `model_name` field is deprecated. Use `model` instead.                 This will be removed in a future release.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "llm_generate |██████████| 25/25 (100.0%) | ⏳ 00:30<00:00 |  1.21s/it"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ Generated 25 queries from dimension tuples\n",
      "\n",
      "📋 Sample tuple → query conversions:\n",
      "\n",
      "Tuple 1: dietary_restriction: gluten-free, cuisine_type: any cuisine, meal_type: dessert, skill_level: beginner\n",
      "Query: '\"Hey there! I'm on the hunt for an easy gluten-free dessert recipe that doesn't stick to any specific cuisine. I’m a beginner in the kitchen, so something simple and delicious would be perfect. Any suggestions?\"'\n",
      "\n",
      "Tuple 2: dietary_restriction: gluten-free, cuisine_type: any cuisine, meal_type: dessert, skill_level: intermediate\n",
      "Query: '\"Hey there! I'm looking for a delicious dessert recipe that’s gluten-free. I’m open to any type of cuisine, but I want something that will challenge my intermediate cooking skills a bit. Could you whip up an exciting recipe suggestion for me?\"'\n",
      "\n",
      "Tuple 3: dietary_restriction: vegan, cuisine_type: American, meal_type: lunch, skill_level: advanced\n",
      "Query: 'I'm looking for an advanced vegan lunch recipe that captures the essence of American cuisine. Something creative and challenging, perhaps a dish that utilizes innovative cooking techniques or unique plant-based ingredients. Any recommendations?'\n",
      "\n",
      "🎯 Created dataset with 25 queries ready for testing!\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "selected_tuples = random.sample(dimension_tuples, 25)\n",
    "\n",
    "print(f\"📝 Selected {len(selected_tuples)} dimension tuples for query generation\")\n",
    "\n",
    "# Step 2: Create dataframe for query generation\n",
    "query_input = []\n",
    "for tuple_data in selected_tuples:\n",
    "    tuple_str = f\"dietary_restriction: {tuple_data['dietary_restriction']}, cuisine_type: {tuple_data['cuisine_type']}, meal_type: {tuple_data['meal_type']}, skill_level: {tuple_data['skill_level']}\"\n",
    "    query_input.append(\n",
    "        {\n",
    "            # 'tuple_id': tuple_data['tuple_id'],\n",
    "            \"tuple_description\": tuple_str,\n",
    "            \"dietary_restriction\": tuple_data[\"dietary_restriction\"],\n",
    "            \"cuisine_type\": tuple_data[\"cuisine_type\"],\n",
    "            \"meal_type\": tuple_data[\"meal_type\"],\n",
    "            \"skill_level\": tuple_data[\"skill_level\"],\n",
    "        }\n",
    "    )\n",
    "\n",
    "query_df = pd.DataFrame(query_input)\n",
    "\n",
    "# Step 3: Template for converting dimension tuples to natural language queries\n",
    "query_template = PromptTemplate(\"\"\"\n",
    "Convert this dimension tuple into a realistic user query for a Recipe Bot:\n",
    "\n",
    "Dimension tuple: {tuple_description}\n",
    "\n",
    "Create a natural language query that a real user with these characteristics might ask. Be creative and vary your style significantly.\n",
    "\n",
    "Vary your vocabulary, sentence structure, and level of detail. Generate 1 unique, realistic query:\n",
    "\"\"\")\n",
    "\n",
    "print(\"🎯 Converting dimension tuples to natural language queries...\")\n",
    "\n",
    "# Step 4: Generate the queries with higher temperature for variety\n",
    "phoenix_model_creative = OpenAIModel(model_name=MODEL_NAME, temperature=0.9)\n",
    "\n",
    "queries_result = llm_generate(\n",
    "    dataframe=query_df, template=query_template, model=phoenix_model_creative\n",
    ")\n",
    "\n",
    "print(f\"✅ Generated {len(queries_result)} queries from dimension tuples\")\n",
    "\n",
    "# Step 5: Show examples of tuple → query conversion\n",
    "print(\"\\n📋 Sample tuple → query conversions:\")\n",
    "for i in range(min(3, len(queries_result))):\n",
    "    input_row = query_df.iloc[i]\n",
    "    query_row = queries_result.iloc[i]\n",
    "    # Clean the query for display too\n",
    "    clean_query = query_row[\"output\"].strip().strip('\"').strip(\"'\").strip()\n",
    "    print(f\"\\nTuple {i + 1}: {input_row['tuple_description']}\")\n",
    "    print(f\"Query: {clean_query}\")\n",
    "\n",
    "# Step 6: Create final dataset with tuple information\n",
    "final_data = []\n",
    "for idx in range(len(queries_result)):\n",
    "    query_row = queries_result.iloc[idx]\n",
    "    original_input = query_df.iloc[idx]\n",
    "\n",
    "    # Clean the query: strip quotes and extra whitespace\n",
    "    clean_query = query_row[\"output\"].strip().strip('\"').strip(\"'\").strip()\n",
    "\n",
    "    final_data.append(\n",
    "        {\n",
    "            \"id\": f\"SYN{idx + 1:03d}\",\n",
    "            \"query\": clean_query,\n",
    "            \"dietary_restriction\": original_input[\"dietary_restriction\"],\n",
    "            \"cuisine_type\": original_input[\"cuisine_type\"],\n",
    "            \"meal_type\": original_input[\"meal_type\"],\n",
    "            \"skill_level\": original_input[\"skill_level\"],\n",
    "            \"tuple_description\": original_input[\"tuple_description\"],\n",
    "        }\n",
    "    )\n",
    "\n",
    "all_queries_df = pd.DataFrame(final_data)\n",
    "print(f\"\\n🎯 Created dataset with {len(all_queries_df)} queries ready for testing!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Quality Review\n",
    "\n",
    "Review the generated queries to make sure they're diverse and realistic: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>query</th>\n",
       "      <th>dietary_restriction</th>\n",
       "      <th>cuisine_type</th>\n",
       "      <th>meal_type</th>\n",
       "      <th>skill_level</th>\n",
       "      <th>tuple_description</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>SYN001</td>\n",
       "      <td>Hey there! I'm on the hunt for an easy gluten-free dessert recipe that doesn't stick to any specific cuisine. I’m a beginner in the kitchen, so something simple and delicious would be perfect. Any suggestions?</td>\n",
       "      <td>gluten-free</td>\n",
       "      <td>any cuisine</td>\n",
       "      <td>dessert</td>\n",
       "      <td>beginner</td>\n",
       "      <td>dietary_restriction: gluten-free, cuisine_type: any cuisine, meal_type: dessert, skill_level: beginner</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>SYN002</td>\n",
       "      <td>Hey there! I'm looking for a delicious dessert recipe that’s gluten-free. I’m open to any type of cuisine, but I want something that will challenge my intermediate cooking skills a bit. Could you whip up an exciting recipe suggestion for me?</td>\n",
       "      <td>gluten-free</td>\n",
       "      <td>any cuisine</td>\n",
       "      <td>dessert</td>\n",
       "      <td>intermediate</td>\n",
       "      <td>dietary_restriction: gluten-free, cuisine_type: any cuisine, meal_type: dessert, skill_level: intermediate</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>SYN003</td>\n",
       "      <td>I'm looking for an advanced vegan lunch recipe that captures the essence of American cuisine. Something creative and challenging, perhaps a dish that utilizes innovative cooking techniques or unique plant-based ingredients. Any recommendations?</td>\n",
       "      <td>vegan</td>\n",
       "      <td>American</td>\n",
       "      <td>lunch</td>\n",
       "      <td>advanced</td>\n",
       "      <td>dietary_restriction: vegan, cuisine_type: American, meal_type: lunch, skill_level: advanced</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>SYN004</td>\n",
       "      <td>I’m looking for an Italian dessert recipe that’s a bit challenging but not too complicated since I have some cooking experience. I have no dietary restrictions, so feel free to suggest something indulgent. Any recommendations?</td>\n",
       "      <td>no restrictions</td>\n",
       "      <td>Italian</td>\n",
       "      <td>dessert</td>\n",
       "      <td>intermediate</td>\n",
       "      <td>dietary_restriction: no restrictions, cuisine_type: Italian, meal_type: dessert, skill_level: intermediate</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>SYN005</td>\n",
       "      <td>I'm looking for an advanced vegan breakfast recipe that has a Mediterranean flair. I've mastered some complex techniques in the kitchen, so I'm eager for a dish that will really challenge my skills and impress my guests. Any suggestions?</td>\n",
       "      <td>vegan</td>\n",
       "      <td>Mediterranean</td>\n",
       "      <td>breakfast</td>\n",
       "      <td>advanced</td>\n",
       "      <td>dietary_restriction: vegan, cuisine_type: Mediterranean, meal_type: breakfast, skill_level: advanced</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>SYN006</td>\n",
       "      <td>Hey there! I'm on the lookout for a vegetarian Asian snack recipe that isn't too beginner-friendly but also not too complicated. I’d love something that strikes a balance and showcases some authentic flavors. Any suggestions?</td>\n",
       "      <td>vegetarian</td>\n",
       "      <td>Asian</td>\n",
       "      <td>snack</td>\n",
       "      <td>intermediate</td>\n",
       "      <td>dietary_restriction: vegetarian, cuisine_type: Asian, meal_type: snack, skill_level: intermediate</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>SYN007</td>\n",
       "      <td>Hey there, Recipe Bot! I'm looking to whip up a simple breakfast with an Asian twist, and I don't have any dietary restrictions. Can you suggest some easy recipes that are perfect for a cooking novice like me? Thanks!</td>\n",
       "      <td>no restrictions</td>\n",
       "      <td>Asian</td>\n",
       "      <td>breakfast</td>\n",
       "      <td>beginner</td>\n",
       "      <td>dietary_restriction: no restrictions, cuisine_type: Asian, meal_type: breakfast, skill_level: beginner</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>SYN008</td>\n",
       "      <td>I'm looking for a delicious vegan Mediterranean dinner recipe that's perfect for someone with an intermediate cooking skill level. Can you suggest something that showcases bold flavors and fresh ingredients? Maybe a dish that would impress my friends at a dinner party?</td>\n",
       "      <td>vegan</td>\n",
       "      <td>Mediterranean</td>\n",
       "      <td>dinner</td>\n",
       "      <td>intermediate</td>\n",
       "      <td>dietary_restriction: vegan, cuisine_type: Mediterranean, meal_type: dinner, skill_level: intermediate</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>SYN009</td>\n",
       "      <td>I'm looking for an intermediate-level gluten-free Asian lunch recipe that I can try out. Do you have any delicious options that use fresh ingredients and maybe a bit of spice? Thanks!</td>\n",
       "      <td>gluten-free</td>\n",
       "      <td>Asian</td>\n",
       "      <td>lunch</td>\n",
       "      <td>intermediate</td>\n",
       "      <td>dietary_restriction: gluten-free, cuisine_type: Asian, meal_type: lunch, skill_level: intermediate</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>SYN010</td>\n",
       "      <td>Hey there! I’m looking for a simple vegan Italian dinner recipe since I’m just starting out in the kitchen. Any recommendations for a dish that won’t overwhelm me but still has that delicious Italian flair? Thanks a bunch!</td>\n",
       "      <td>vegan</td>\n",
       "      <td>Italian</td>\n",
       "      <td>dinner</td>\n",
       "      <td>beginner</td>\n",
       "      <td>dietary_restriction: vegan, cuisine_type: Italian, meal_type: dinner, skill_level: beginner</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>SYN011</td>\n",
       "      <td>Hey there! I'm looking to whip up a delicious gluten-free dessert with a Mexican twist. I’m pretty comfortable in the kitchen, so I’d love something that’s a bit of a challenge but not too overwhelming. Could you suggest a recipe that fits the bill? Thanks!</td>\n",
       "      <td>gluten-free</td>\n",
       "      <td>Mexican</td>\n",
       "      <td>dessert</td>\n",
       "      <td>intermediate</td>\n",
       "      <td>dietary_restriction: gluten-free, cuisine_type: Mexican, meal_type: dessert, skill_level: intermediate</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>SYN012</td>\n",
       "      <td>I'm looking for a challenging vegan snack recipe that spans different cuisines. I want something that will really test my culinary skills and creativity in the kitchen. Any suggestions?</td>\n",
       "      <td>vegan</td>\n",
       "      <td>any cuisine</td>\n",
       "      <td>snack</td>\n",
       "      <td>advanced</td>\n",
       "      <td>dietary_restriction: vegan, cuisine_type: any cuisine, meal_type: snack, skill_level: advanced</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>SYN013</td>\n",
       "      <td>Hey Recipe Bot! I'm looking for an advanced vegetarian breakfast recipe that’s inspired by Asian cuisine. I want something that’s not only delicious but also a bit of a culinary challenge. Any ideas on how I can impress my taste buds and showcase my cooking skills?</td>\n",
       "      <td>vegetarian</td>\n",
       "      <td>Asian</td>\n",
       "      <td>breakfast</td>\n",
       "      <td>advanced</td>\n",
       "      <td>dietary_restriction: vegetarian, cuisine_type: Asian, meal_type: breakfast, skill_level: advanced</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>SYN014</td>\n",
       "      <td>I'm looking to impress my dinner guests with an advanced vegetarian dish that showcases flavors from any cuisine. Could you suggest a recipe that really pushes my culinary skills?</td>\n",
       "      <td>vegetarian</td>\n",
       "      <td>any cuisine</td>\n",
       "      <td>dinner</td>\n",
       "      <td>advanced</td>\n",
       "      <td>dietary_restriction: vegetarian, cuisine_type: any cuisine, meal_type: dinner, skill_level: advanced</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>SYN015</td>\n",
       "      <td>Hey there, Recipe Bot! I'm looking to whip up a delicious Asian dinner that’s gluten-free and requires a bit of skill in the kitchen. Could you suggest a few intermediate recipes that will impress my guests? I’d love something flavorful and authentic!</td>\n",
       "      <td>gluten-free</td>\n",
       "      <td>Asian</td>\n",
       "      <td>dinner</td>\n",
       "      <td>intermediate</td>\n",
       "      <td>dietary_restriction: gluten-free, cuisine_type: Asian, meal_type: dinner, skill_level: intermediate</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>SYN016</td>\n",
       "      <td>Hey there, I'm looking for a simple and delicious American dessert recipe that’s vegetarian-friendly. I’m a beginner in the kitchen, so if you have any straightforward ideas that don’t require fancy techniques or hard-to-find ingredients, I’d love to hear them!</td>\n",
       "      <td>vegetarian</td>\n",
       "      <td>American</td>\n",
       "      <td>dessert</td>\n",
       "      <td>beginner</td>\n",
       "      <td>dietary_restriction: vegetarian, cuisine_type: American, meal_type: dessert, skill_level: beginner</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>SYN017</td>\n",
       "      <td>Hey there! I'm feeling adventurous and would love to whip up an intermediate-level Asian dessert. I have no dietary restrictions, so I'm open to trying something new and exciting. Do you have any delicious recipe suggestions that fit the bill?</td>\n",
       "      <td>no restrictions</td>\n",
       "      <td>Asian</td>\n",
       "      <td>dessert</td>\n",
       "      <td>intermediate</td>\n",
       "      <td>dietary_restriction: no restrictions, cuisine_type: Asian, meal_type: dessert, skill_level: intermediate</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>SYN018</td>\n",
       "      <td>Hi there! I'm looking for some easy lunch ideas that are Italian but totally gluten-free. I’m a beginner in the kitchen, so I’d appreciate simple recipes that don’t require too many fancy ingredients or complicated steps. Any suggestions?</td>\n",
       "      <td>gluten-free</td>\n",
       "      <td>Italian</td>\n",
       "      <td>lunch</td>\n",
       "      <td>beginner</td>\n",
       "      <td>dietary_restriction: gluten-free, cuisine_type: Italian, meal_type: lunch, skill_level: beginner</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>SYN019</td>\n",
       "      <td>Hey there! I’m looking for some easy vegetarian dinner recipes with an American flair. I’m a beginner in the kitchen, so if you could suggest something simple to whip up, that would be awesome. Thanks!</td>\n",
       "      <td>vegetarian</td>\n",
       "      <td>American</td>\n",
       "      <td>dinner</td>\n",
       "      <td>beginner</td>\n",
       "      <td>dietary_restriction: vegetarian, cuisine_type: American, meal_type: dinner, skill_level: beginner</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>SYN020</td>\n",
       "      <td>Hey there! I'm looking for some easy Italian breakfast recipes that are completely vegan. I'm a beginner in the kitchen, so I need something straightforward. Any tasty suggestions?</td>\n",
       "      <td>vegan</td>\n",
       "      <td>Italian</td>\n",
       "      <td>breakfast</td>\n",
       "      <td>beginner</td>\n",
       "      <td>dietary_restriction: vegan, cuisine_type: Italian, meal_type: breakfast, skill_level: beginner</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>SYN021</td>\n",
       "      <td>Hey there! I'm looking for an intermediate-level vegan snack recipe that has an Asian flair. Any suggestions that would satisfy my cravings while sticking to my dietary preferences?</td>\n",
       "      <td>vegan</td>\n",
       "      <td>Asian</td>\n",
       "      <td>snack</td>\n",
       "      <td>intermediate</td>\n",
       "      <td>dietary_restriction: vegan, cuisine_type: Asian, meal_type: snack, skill_level: intermediate</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>SYN022</td>\n",
       "      <td>Hi there! I'm looking for a simple gluten-free dinner recipe that has a Mediterranean flair. I’m just starting out in the kitchen, so anything that’s beginner-friendly would be fantastic. Any suggestions?</td>\n",
       "      <td>gluten-free</td>\n",
       "      <td>Mediterranean</td>\n",
       "      <td>dinner</td>\n",
       "      <td>beginner</td>\n",
       "      <td>dietary_restriction: gluten-free, cuisine_type: Mediterranean, meal_type: dinner, skill_level: beginner</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>SYN023</td>\n",
       "      <td>Hey there! I'm looking for a beginner-friendly vegetarian Mexican breakfast recipe. Something simple but delicious that I can whip up in the morning. Any ideas?</td>\n",
       "      <td>vegetarian</td>\n",
       "      <td>Mexican</td>\n",
       "      <td>breakfast</td>\n",
       "      <td>beginner</td>\n",
       "      <td>dietary_restriction: vegetarian, cuisine_type: Mexican, meal_type: breakfast, skill_level: beginner</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>SYN024</td>\n",
       "      <td>Hi there! I'm a beginner in the kitchen and really want to whip up a tasty vegan snack with an Italian twist. Do you have any simple recipes for me that fit this vibe? Thanks!</td>\n",
       "      <td>vegan</td>\n",
       "      <td>Italian</td>\n",
       "      <td>snack</td>\n",
       "      <td>beginner</td>\n",
       "      <td>dietary_restriction: vegan, cuisine_type: Italian, meal_type: snack, skill_level: beginner</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>SYN025</td>\n",
       "      <td>I'm looking for a delicious dessert recipe that aligns with my vegetarian lifestyle and has a Mediterranean flair. I have some experience in the kitchen, so I'm ready for an intermediate-level challenge. Can you recommend something that combines traditional flavors with a modern twist?</td>\n",
       "      <td>vegetarian</td>\n",
       "      <td>Mediterranean</td>\n",
       "      <td>dessert</td>\n",
       "      <td>intermediate</td>\n",
       "      <td>dietary_restriction: vegetarian, cuisine_type: Mediterranean, meal_type: dessert, skill_level: intermediate</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        id  \\\n",
       "0   SYN001   \n",
       "1   SYN002   \n",
       "2   SYN003   \n",
       "3   SYN004   \n",
       "4   SYN005   \n",
       "5   SYN006   \n",
       "6   SYN007   \n",
       "7   SYN008   \n",
       "8   SYN009   \n",
       "9   SYN010   \n",
       "10  SYN011   \n",
       "11  SYN012   \n",
       "12  SYN013   \n",
       "13  SYN014   \n",
       "14  SYN015   \n",
       "15  SYN016   \n",
       "16  SYN017   \n",
       "17  SYN018   \n",
       "18  SYN019   \n",
       "19  SYN020   \n",
       "20  SYN021   \n",
       "21  SYN022   \n",
       "22  SYN023   \n",
       "23  SYN024   \n",
       "24  SYN025   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                             query  \\\n",
       "0                                                                                Hey there! I'm on the hunt for an easy gluten-free dessert recipe that doesn't stick to any specific cuisine. I’m a beginner in the kitchen, so something simple and delicious would be perfect. Any suggestions?   \n",
       "1                                                Hey there! I'm looking for a delicious dessert recipe that’s gluten-free. I’m open to any type of cuisine, but I want something that will challenge my intermediate cooking skills a bit. Could you whip up an exciting recipe suggestion for me?   \n",
       "2                                             I'm looking for an advanced vegan lunch recipe that captures the essence of American cuisine. Something creative and challenging, perhaps a dish that utilizes innovative cooking techniques or unique plant-based ingredients. Any recommendations?   \n",
       "3                                                               I’m looking for an Italian dessert recipe that’s a bit challenging but not too complicated since I have some cooking experience. I have no dietary restrictions, so feel free to suggest something indulgent. Any recommendations?   \n",
       "4                                                    I'm looking for an advanced vegan breakfast recipe that has a Mediterranean flair. I've mastered some complex techniques in the kitchen, so I'm eager for a dish that will really challenge my skills and impress my guests. Any suggestions?   \n",
       "5                                                                Hey there! I'm on the lookout for a vegetarian Asian snack recipe that isn't too beginner-friendly but also not too complicated. I’d love something that strikes a balance and showcases some authentic flavors. Any suggestions?   \n",
       "6                                                                        Hey there, Recipe Bot! I'm looking to whip up a simple breakfast with an Asian twist, and I don't have any dietary restrictions. Can you suggest some easy recipes that are perfect for a cooking novice like me? Thanks!   \n",
       "7                    I'm looking for a delicious vegan Mediterranean dinner recipe that's perfect for someone with an intermediate cooking skill level. Can you suggest something that showcases bold flavors and fresh ingredients? Maybe a dish that would impress my friends at a dinner party?   \n",
       "8                                                                                                          I'm looking for an intermediate-level gluten-free Asian lunch recipe that I can try out. Do you have any delicious options that use fresh ingredients and maybe a bit of spice? Thanks!   \n",
       "9                                                                   Hey there! I’m looking for a simple vegan Italian dinner recipe since I’m just starting out in the kitchen. Any recommendations for a dish that won’t overwhelm me but still has that delicious Italian flair? Thanks a bunch!   \n",
       "10                               Hey there! I'm looking to whip up a delicious gluten-free dessert with a Mexican twist. I’m pretty comfortable in the kitchen, so I’d love something that’s a bit of a challenge but not too overwhelming. Could you suggest a recipe that fits the bill? Thanks!   \n",
       "11                                                                                                       I'm looking for a challenging vegan snack recipe that spans different cuisines. I want something that will really test my culinary skills and creativity in the kitchen. Any suggestions?   \n",
       "12                       Hey Recipe Bot! I'm looking for an advanced vegetarian breakfast recipe that’s inspired by Asian cuisine. I want something that’s not only delicious but also a bit of a culinary challenge. Any ideas on how I can impress my taste buds and showcase my cooking skills?   \n",
       "13                                                                                                             I'm looking to impress my dinner guests with an advanced vegetarian dish that showcases flavors from any cuisine. Could you suggest a recipe that really pushes my culinary skills?   \n",
       "14                                     Hey there, Recipe Bot! I'm looking to whip up a delicious Asian dinner that’s gluten-free and requires a bit of skill in the kitchen. Could you suggest a few intermediate recipes that will impress my guests? I’d love something flavorful and authentic!   \n",
       "15                           Hey there, I'm looking for a simple and delicious American dessert recipe that’s vegetarian-friendly. I’m a beginner in the kitchen, so if you have any straightforward ideas that don’t require fancy techniques or hard-to-find ingredients, I’d love to hear them!   \n",
       "16                                             Hey there! I'm feeling adventurous and would love to whip up an intermediate-level Asian dessert. I have no dietary restrictions, so I'm open to trying something new and exciting. Do you have any delicious recipe suggestions that fit the bill?   \n",
       "17                                                  Hi there! I'm looking for some easy lunch ideas that are Italian but totally gluten-free. I’m a beginner in the kitchen, so I’d appreciate simple recipes that don’t require too many fancy ingredients or complicated steps. Any suggestions?   \n",
       "18                                                                                       Hey there! I’m looking for some easy vegetarian dinner recipes with an American flair. I’m a beginner in the kitchen, so if you could suggest something simple to whip up, that would be awesome. Thanks!   \n",
       "19                                                                                                            Hey there! I'm looking for some easy Italian breakfast recipes that are completely vegan. I'm a beginner in the kitchen, so I need something straightforward. Any tasty suggestions?   \n",
       "20                                                                                                           Hey there! I'm looking for an intermediate-level vegan snack recipe that has an Asian flair. Any suggestions that would satisfy my cravings while sticking to my dietary preferences?   \n",
       "21                                                                                    Hi there! I'm looking for a simple gluten-free dinner recipe that has a Mediterranean flair. I’m just starting out in the kitchen, so anything that’s beginner-friendly would be fantastic. Any suggestions?   \n",
       "22                                                                                                                                Hey there! I'm looking for a beginner-friendly vegetarian Mexican breakfast recipe. Something simple but delicious that I can whip up in the morning. Any ideas?   \n",
       "23                                                                                                                 Hi there! I'm a beginner in the kitchen and really want to whip up a tasty vegan snack with an Italian twist. Do you have any simple recipes for me that fit this vibe? Thanks!   \n",
       "24  I'm looking for a delicious dessert recipe that aligns with my vegetarian lifestyle and has a Mediterranean flair. I have some experience in the kitchen, so I'm ready for an intermediate-level challenge. Can you recommend something that combines traditional flavors with a modern twist?   \n",
       "\n",
       "   dietary_restriction   cuisine_type  meal_type   skill_level  \\\n",
       "0          gluten-free    any cuisine    dessert      beginner   \n",
       "1          gluten-free    any cuisine    dessert  intermediate   \n",
       "2                vegan       American      lunch      advanced   \n",
       "3      no restrictions        Italian    dessert  intermediate   \n",
       "4                vegan  Mediterranean  breakfast      advanced   \n",
       "5           vegetarian          Asian      snack  intermediate   \n",
       "6      no restrictions          Asian  breakfast      beginner   \n",
       "7                vegan  Mediterranean     dinner  intermediate   \n",
       "8          gluten-free          Asian      lunch  intermediate   \n",
       "9                vegan        Italian     dinner      beginner   \n",
       "10         gluten-free        Mexican    dessert  intermediate   \n",
       "11               vegan    any cuisine      snack      advanced   \n",
       "12          vegetarian          Asian  breakfast      advanced   \n",
       "13          vegetarian    any cuisine     dinner      advanced   \n",
       "14         gluten-free          Asian     dinner  intermediate   \n",
       "15          vegetarian       American    dessert      beginner   \n",
       "16     no restrictions          Asian    dessert  intermediate   \n",
       "17         gluten-free        Italian      lunch      beginner   \n",
       "18          vegetarian       American     dinner      beginner   \n",
       "19               vegan        Italian  breakfast      beginner   \n",
       "20               vegan          Asian      snack  intermediate   \n",
       "21         gluten-free  Mediterranean     dinner      beginner   \n",
       "22          vegetarian        Mexican  breakfast      beginner   \n",
       "23               vegan        Italian      snack      beginner   \n",
       "24          vegetarian  Mediterranean    dessert  intermediate   \n",
       "\n",
       "                                                                                              tuple_description  \n",
       "0        dietary_restriction: gluten-free, cuisine_type: any cuisine, meal_type: dessert, skill_level: beginner  \n",
       "1    dietary_restriction: gluten-free, cuisine_type: any cuisine, meal_type: dessert, skill_level: intermediate  \n",
       "2                   dietary_restriction: vegan, cuisine_type: American, meal_type: lunch, skill_level: advanced  \n",
       "3    dietary_restriction: no restrictions, cuisine_type: Italian, meal_type: dessert, skill_level: intermediate  \n",
       "4          dietary_restriction: vegan, cuisine_type: Mediterranean, meal_type: breakfast, skill_level: advanced  \n",
       "5             dietary_restriction: vegetarian, cuisine_type: Asian, meal_type: snack, skill_level: intermediate  \n",
       "6        dietary_restriction: no restrictions, cuisine_type: Asian, meal_type: breakfast, skill_level: beginner  \n",
       "7         dietary_restriction: vegan, cuisine_type: Mediterranean, meal_type: dinner, skill_level: intermediate  \n",
       "8            dietary_restriction: gluten-free, cuisine_type: Asian, meal_type: lunch, skill_level: intermediate  \n",
       "9                   dietary_restriction: vegan, cuisine_type: Italian, meal_type: dinner, skill_level: beginner  \n",
       "10       dietary_restriction: gluten-free, cuisine_type: Mexican, meal_type: dessert, skill_level: intermediate  \n",
       "11               dietary_restriction: vegan, cuisine_type: any cuisine, meal_type: snack, skill_level: advanced  \n",
       "12            dietary_restriction: vegetarian, cuisine_type: Asian, meal_type: breakfast, skill_level: advanced  \n",
       "13         dietary_restriction: vegetarian, cuisine_type: any cuisine, meal_type: dinner, skill_level: advanced  \n",
       "14          dietary_restriction: gluten-free, cuisine_type: Asian, meal_type: dinner, skill_level: intermediate  \n",
       "15           dietary_restriction: vegetarian, cuisine_type: American, meal_type: dessert, skill_level: beginner  \n",
       "16     dietary_restriction: no restrictions, cuisine_type: Asian, meal_type: dessert, skill_level: intermediate  \n",
       "17             dietary_restriction: gluten-free, cuisine_type: Italian, meal_type: lunch, skill_level: beginner  \n",
       "18            dietary_restriction: vegetarian, cuisine_type: American, meal_type: dinner, skill_level: beginner  \n",
       "19               dietary_restriction: vegan, cuisine_type: Italian, meal_type: breakfast, skill_level: beginner  \n",
       "20                 dietary_restriction: vegan, cuisine_type: Asian, meal_type: snack, skill_level: intermediate  \n",
       "21      dietary_restriction: gluten-free, cuisine_type: Mediterranean, meal_type: dinner, skill_level: beginner  \n",
       "22          dietary_restriction: vegetarian, cuisine_type: Mexican, meal_type: breakfast, skill_level: beginner  \n",
       "23                   dietary_restriction: vegan, cuisine_type: Italian, meal_type: snack, skill_level: beginner  \n",
       "24  dietary_restriction: vegetarian, cuisine_type: Mediterranean, meal_type: dessert, skill_level: intermediate  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Display all rows and columns, and show full text in each cell for all_queries_df\n",
    "with pd.option_context(\n",
    "    \"display.max_rows\", None, \"display.max_columns\", None, \"display.max_colwidth\", None\n",
    "):\n",
    "    display(all_queries_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save Dataset\n",
    "\n",
    "Save the dataset for testing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "💾 Saved dataset to: data/generated_synthetic_queries.csv\n",
      "📊 Ready for testing with 25 queries!\n"
     ]
    }
   ],
   "source": [
    "# Save the dataset to CSV for easy use\n",
    "output_path = OUTPUT_DIR / \"generated_synthetic_queries.csv\"\n",
    "all_queries_df.to_csv(output_path, index=False)\n",
    "\n",
    "print(f\"💾 Saved dataset to: {output_path}\")\n",
    "print(f\"📊 Ready for testing with {len(all_queries_df)} queries!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Upload to Phoenix\n",
    "\n",
    "You can either:\n",
    "- **Option A:** Manually upload the CSV file to Phoenix UI\n",
    "- **Option B:** Use the SDK upload below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "📤 Uploading dataset...\n",
      "💾 Examples uploaded: http://127.0.0.1:6006/datasets/RGF0YXNldDox/examples\n",
      "🗄️ Dataset version ID: RGF0YXNldFZlcnNpb246MQ==\n"
     ]
    }
   ],
   "source": [
    "# original_test = pd.read_csv(\"/Users/sallyanndelucia/Documents/GitHub/recipe-chatbot/data/generated_synthetic_queries.csv\")\n",
    "px_client = AsyncClient()\n",
    "dataset = await px_client.datasets.create_dataset(\n",
    "    dataframe=all_queries_df,\n",
    "    name=\"recipe-bot-synthetic-queries\",\n",
    "    input_keys=[\"query\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "raw"
    }
   },
   "source": [
    "## Part 1 Complete ✅\n",
    "\n",
    "**What you now have:**\n",
    "- 25 diverse test queries saved as CSV\n",
    "- Dataset uploaded to Phoenix (ready for testing)\n",
    "- Systematic coverage across key user dimensions  \n",
    "\n",
    "**Next steps:**\n",
    "1. Go to Phoenix UI\n",
    "2. Run your Recipe Bot on these queries\n",
    "3. Annotate problems you find\n",
    "4. Come back to this notebook for analysis\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "raw"
    }
   },
   "source": [
    "# Part 2: Initial Error Analysis\n",
    "\n",
    "## Step 2.1: Run Bot on Synthetic Queries\n",
    "\n",
    "1. **Upload Dataset**: Load your synthetic queries into Phoenix playground\n",
    "2. **Configure Bot**: Import your Recipe Bot prompt \n",
    "3. **Run Tests**: Execute all queries through your bot\n",
    "4. **Record Results**: Save the interaction traces\n",
    "\n",
    "## Step 2.2: Open Coding\n",
    "\n",
    "Review the recorded traces and perform open coding to identify themes, patterns, and potential errors in your bot's responses.\n",
    "\n",
    "**What to look for:**\n",
    "- Factual errors or incorrect recommendations\n",
    "- Confusing or unhelpful responses\n",
    "- Inconsistent behavior across similar queries\n",
    "- Format and communication issues\n",
    "\n",
    "**How to annotate:**\n",
    "- Be specific about what went wrong\n",
    "- Note why something is problematic for users \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "raw"
    }
   },
   "source": [
    "# Part 3: Axial Coding & Taxonomy Definition\n",
    "\n",
    "## Step 3.1: Export Annotated Traces\n",
    "\n",
    "Export your annotated traces and annotations from Phoenix.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>span_kind</th>\n",
       "      <th>parent_id</th>\n",
       "      <th>start_time</th>\n",
       "      <th>end_time</th>\n",
       "      <th>status_code</th>\n",
       "      <th>status_message</th>\n",
       "      <th>events</th>\n",
       "      <th>context.span_id</th>\n",
       "      <th>context.trace_id</th>\n",
       "      <th>...</th>\n",
       "      <th>attributes.llm.token_count.prompt_details.cache_read</th>\n",
       "      <th>attributes.llm.token_count.prompt</th>\n",
       "      <th>attributes.llm.invocation_parameters</th>\n",
       "      <th>attributes.url</th>\n",
       "      <th>attributes.llm.token_count.completion</th>\n",
       "      <th>attributes.input.mime_type</th>\n",
       "      <th>attributes.metadata</th>\n",
       "      <th>attributes.llm.token_count.prompt_details.audio</th>\n",
       "      <th>attributes.output.mime_type</th>\n",
       "      <th>attributes.llm.system</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ChatCompletion</td>\n",
       "      <td>LLM</td>\n",
       "      <td>None</td>\n",
       "      <td>2025-07-30 05:02:54.680996+00:00</td>\n",
       "      <td>2025-07-30 05:03:06.042090+00:00</td>\n",
       "      <td>OK</td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>16e2eccbb4fd49ca</td>\n",
       "      <td>1800389113fe83f890dfc1c81c7994ce</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>397</td>\n",
       "      <td>{\"temperature\": 1.0, \"top_p\": 1.0}</td>\n",
       "      <td>{'path': 'chat/completions', 'full': 'https://...</td>\n",
       "      <td>387</td>\n",
       "      <td>application/json</td>\n",
       "      <td>{'phoenix_prompt_id': 'recipe-assistant-v1-tes...</td>\n",
       "      <td>0</td>\n",
       "      <td>text/plain</td>\n",
       "      <td>openai</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ChatCompletion</td>\n",
       "      <td>LLM</td>\n",
       "      <td>None</td>\n",
       "      <td>2025-07-30 05:02:54.681153+00:00</td>\n",
       "      <td>2025-07-30 05:03:08.212350+00:00</td>\n",
       "      <td>OK</td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>7138c4e0fffc1305</td>\n",
       "      <td>bce5739e6fda4b5a8e4828c13e0b0a5c</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>388</td>\n",
       "      <td>{\"temperature\": 1.0, \"top_p\": 1.0}</td>\n",
       "      <td>{'path': 'chat/completions', 'full': 'https://...</td>\n",
       "      <td>456</td>\n",
       "      <td>application/json</td>\n",
       "      <td>{'phoenix_prompt_id': 'recipe-assistant-v1-tes...</td>\n",
       "      <td>0</td>\n",
       "      <td>text/plain</td>\n",
       "      <td>openai</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ChatCompletion</td>\n",
       "      <td>LLM</td>\n",
       "      <td>None</td>\n",
       "      <td>2025-07-30 05:02:54.680560+00:00</td>\n",
       "      <td>2025-07-30 05:03:11.375935+00:00</td>\n",
       "      <td>OK</td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>41565909b57f1b2b</td>\n",
       "      <td>32e80f33cf5594698b13c7aa19a30dbb</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>407</td>\n",
       "      <td>{\"temperature\": 1.0, \"top_p\": 1.0}</td>\n",
       "      <td>{'path': 'chat/completions', 'full': 'https://...</td>\n",
       "      <td>666</td>\n",
       "      <td>application/json</td>\n",
       "      <td>{'phoenix_prompt_id': 'recipe-assistant-v1-tes...</td>\n",
       "      <td>0</td>\n",
       "      <td>text/plain</td>\n",
       "      <td>openai</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ChatCompletion</td>\n",
       "      <td>LLM</td>\n",
       "      <td>None</td>\n",
       "      <td>2025-07-30 05:03:06.057379+00:00</td>\n",
       "      <td>2025-07-30 05:03:18.969275+00:00</td>\n",
       "      <td>OK</td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>26cf736bceaea3dc</td>\n",
       "      <td>b8cc683e595ef3748ac3b05f93f09fb4</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>398</td>\n",
       "      <td>{\"temperature\": 1.0, \"top_p\": 1.0}</td>\n",
       "      <td>{'path': 'chat/completions', 'full': 'https://...</td>\n",
       "      <td>473</td>\n",
       "      <td>application/json</td>\n",
       "      <td>{'phoenix_prompt_id': 'recipe-assistant-v1-tes...</td>\n",
       "      <td>0</td>\n",
       "      <td>text/plain</td>\n",
       "      <td>openai</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ChatCompletion</td>\n",
       "      <td>LLM</td>\n",
       "      <td>None</td>\n",
       "      <td>2025-07-30 05:03:08.213397+00:00</td>\n",
       "      <td>2025-07-30 05:03:19.429567+00:00</td>\n",
       "      <td>OK</td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>ad65c522d55926b7</td>\n",
       "      <td>f3b8ca911c8fa652aaab22fc01676190</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>390</td>\n",
       "      <td>{\"temperature\": 1.0, \"top_p\": 1.0}</td>\n",
       "      <td>{'path': 'chat/completions', 'full': 'https://...</td>\n",
       "      <td>468</td>\n",
       "      <td>application/json</td>\n",
       "      <td>{'phoenix_prompt_id': 'recipe-assistant-v1-tes...</td>\n",
       "      <td>0</td>\n",
       "      <td>text/plain</td>\n",
       "      <td>openai</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 31 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             name span_kind parent_id                       start_time  \\\n",
       "0  ChatCompletion       LLM      None 2025-07-30 05:02:54.680996+00:00   \n",
       "1  ChatCompletion       LLM      None 2025-07-30 05:02:54.681153+00:00   \n",
       "2  ChatCompletion       LLM      None 2025-07-30 05:02:54.680560+00:00   \n",
       "3  ChatCompletion       LLM      None 2025-07-30 05:03:06.057379+00:00   \n",
       "4  ChatCompletion       LLM      None 2025-07-30 05:03:08.213397+00:00   \n",
       "\n",
       "                          end_time status_code status_message events  \\\n",
       "0 2025-07-30 05:03:06.042090+00:00          OK                    []   \n",
       "1 2025-07-30 05:03:08.212350+00:00          OK                    []   \n",
       "2 2025-07-30 05:03:11.375935+00:00          OK                    []   \n",
       "3 2025-07-30 05:03:18.969275+00:00          OK                    []   \n",
       "4 2025-07-30 05:03:19.429567+00:00          OK                    []   \n",
       "\n",
       "    context.span_id                  context.trace_id  ...  \\\n",
       "0  16e2eccbb4fd49ca  1800389113fe83f890dfc1c81c7994ce  ...   \n",
       "1  7138c4e0fffc1305  bce5739e6fda4b5a8e4828c13e0b0a5c  ...   \n",
       "2  41565909b57f1b2b  32e80f33cf5594698b13c7aa19a30dbb  ...   \n",
       "3  26cf736bceaea3dc  b8cc683e595ef3748ac3b05f93f09fb4  ...   \n",
       "4  ad65c522d55926b7  f3b8ca911c8fa652aaab22fc01676190  ...   \n",
       "\n",
       "  attributes.llm.token_count.prompt_details.cache_read  \\\n",
       "0                                                  0     \n",
       "1                                                  0     \n",
       "2                                                  0     \n",
       "3                                                  0     \n",
       "4                                                  0     \n",
       "\n",
       "  attributes.llm.token_count.prompt attributes.llm.invocation_parameters  \\\n",
       "0                               397   {\"temperature\": 1.0, \"top_p\": 1.0}   \n",
       "1                               388   {\"temperature\": 1.0, \"top_p\": 1.0}   \n",
       "2                               407   {\"temperature\": 1.0, \"top_p\": 1.0}   \n",
       "3                               398   {\"temperature\": 1.0, \"top_p\": 1.0}   \n",
       "4                               390   {\"temperature\": 1.0, \"top_p\": 1.0}   \n",
       "\n",
       "                                      attributes.url  \\\n",
       "0  {'path': 'chat/completions', 'full': 'https://...   \n",
       "1  {'path': 'chat/completions', 'full': 'https://...   \n",
       "2  {'path': 'chat/completions', 'full': 'https://...   \n",
       "3  {'path': 'chat/completions', 'full': 'https://...   \n",
       "4  {'path': 'chat/completions', 'full': 'https://...   \n",
       "\n",
       "   attributes.llm.token_count.completion attributes.input.mime_type  \\\n",
       "0                                    387           application/json   \n",
       "1                                    456           application/json   \n",
       "2                                    666           application/json   \n",
       "3                                    473           application/json   \n",
       "4                                    468           application/json   \n",
       "\n",
       "                                 attributes.metadata  \\\n",
       "0  {'phoenix_prompt_id': 'recipe-assistant-v1-tes...   \n",
       "1  {'phoenix_prompt_id': 'recipe-assistant-v1-tes...   \n",
       "2  {'phoenix_prompt_id': 'recipe-assistant-v1-tes...   \n",
       "3  {'phoenix_prompt_id': 'recipe-assistant-v1-tes...   \n",
       "4  {'phoenix_prompt_id': 'recipe-assistant-v1-tes...   \n",
       "\n",
       "  attributes.llm.token_count.prompt_details.audio attributes.output.mime_type  \\\n",
       "0                                               0                  text/plain   \n",
       "1                                               0                  text/plain   \n",
       "2                                               0                  text/plain   \n",
       "3                                               0                  text/plain   \n",
       "4                                               0                  text/plain   \n",
       "\n",
       "  attributes.llm.system  \n",
       "0                openai  \n",
       "1                openai  \n",
       "2                openai  \n",
       "3                openai  \n",
       "4                openai  \n",
       "\n",
       "[5 rows x 31 columns]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# This method returns a list of dictionaries instead of a DataFrame\n",
    "px_client = AsyncClient()\n",
    "\n",
    "# Query for spans that have notes\n",
    "# query = SpanQuery().where(\"annotations['note']\")\n",
    "spans = await px_client.spans.get_spans_dataframe(\n",
    "    # query=query,\n",
    "    project_identifier=\"UHJvamVjdDoy\"\n",
    ")\n",
    "\n",
    "spans.reset_index(drop=True, inplace=True)\n",
    "\n",
    "spans.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>span_id</th>\n",
       "      <th>annotation_name</th>\n",
       "      <th>annotator_kind</th>\n",
       "      <th>metadata</th>\n",
       "      <th>identifier</th>\n",
       "      <th>id</th>\n",
       "      <th>created_at</th>\n",
       "      <th>updated_at</th>\n",
       "      <th>source</th>\n",
       "      <th>user_id</th>\n",
       "      <th>result.label</th>\n",
       "      <th>result.score</th>\n",
       "      <th>result.explanation</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4df612fcfe75b0b6</td>\n",
       "      <td>note</td>\n",
       "      <td>HUMAN</td>\n",
       "      <td>{}</td>\n",
       "      <td>px-span-note:2025-07-30T00:20:28.839601</td>\n",
       "      <td>U3BhbkFubm90YXRpb246OQ==</td>\n",
       "      <td>2025-07-30T06:20:28+00:00</td>\n",
       "      <td>2025-07-30T06:20:28+00:00</td>\n",
       "      <td>APP</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Ignores dietary restrictions</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>a95eb1dbcd5e729e</td>\n",
       "      <td>note</td>\n",
       "      <td>HUMAN</td>\n",
       "      <td>{}</td>\n",
       "      <td>px-span-note:2025-07-29T23:18:36.864250</td>\n",
       "      <td>U3BhbkFubm90YXRpb246Ng==</td>\n",
       "      <td>2025-07-30T05:18:36+00:00</td>\n",
       "      <td>2025-07-30T05:18:36+00:00</td>\n",
       "      <td>APP</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Incorrect formatting</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3ef7a57bae611fbf</td>\n",
       "      <td>note</td>\n",
       "      <td>HUMAN</td>\n",
       "      <td>{}</td>\n",
       "      <td>px-span-note:2025-07-29T23:16:40.973211</td>\n",
       "      <td>U3BhbkFubm90YXRpb246NA==</td>\n",
       "      <td>2025-07-30T05:16:40+00:00</td>\n",
       "      <td>2025-07-30T05:16:40+00:00</td>\n",
       "      <td>APP</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Recipe provided is a bit complicated for the u...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6587e423ae0948ab</td>\n",
       "      <td>note</td>\n",
       "      <td>HUMAN</td>\n",
       "      <td>{}</td>\n",
       "      <td>px-span-note:2025-07-29T23:09:10.677786</td>\n",
       "      <td>U3BhbkFubm90YXRpb246Mg==</td>\n",
       "      <td>2025-07-30T05:09:10+00:00</td>\n",
       "      <td>2025-07-30T05:09:10+00:00</td>\n",
       "      <td>APP</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Asks for breakfast but a lunch or dinner is su...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4e6a458fbd57e2d4</td>\n",
       "      <td>note</td>\n",
       "      <td>HUMAN</td>\n",
       "      <td>{}</td>\n",
       "      <td>px-span-note:2025-07-29T23:20:36.659177</td>\n",
       "      <td>U3BhbkFubm90YXRpb246OA==</td>\n",
       "      <td>2025-07-30T05:20:36+00:00</td>\n",
       "      <td>2025-07-30T05:20:36+00:00</td>\n",
       "      <td>APP</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Does not include unique ingredients like the u...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            span_id annotation_name annotator_kind metadata  \\\n",
       "0  4df612fcfe75b0b6            note          HUMAN       {}   \n",
       "1  a95eb1dbcd5e729e            note          HUMAN       {}   \n",
       "2  3ef7a57bae611fbf            note          HUMAN       {}   \n",
       "3  6587e423ae0948ab            note          HUMAN       {}   \n",
       "4  4e6a458fbd57e2d4            note          HUMAN       {}   \n",
       "\n",
       "                                identifier                        id  \\\n",
       "0  px-span-note:2025-07-30T00:20:28.839601  U3BhbkFubm90YXRpb246OQ==   \n",
       "1  px-span-note:2025-07-29T23:18:36.864250  U3BhbkFubm90YXRpb246Ng==   \n",
       "2  px-span-note:2025-07-29T23:16:40.973211  U3BhbkFubm90YXRpb246NA==   \n",
       "3  px-span-note:2025-07-29T23:09:10.677786  U3BhbkFubm90YXRpb246Mg==   \n",
       "4  px-span-note:2025-07-29T23:20:36.659177  U3BhbkFubm90YXRpb246OA==   \n",
       "\n",
       "                  created_at                 updated_at source user_id  \\\n",
       "0  2025-07-30T06:20:28+00:00  2025-07-30T06:20:28+00:00    APP    None   \n",
       "1  2025-07-30T05:18:36+00:00  2025-07-30T05:18:36+00:00    APP    None   \n",
       "2  2025-07-30T05:16:40+00:00  2025-07-30T05:16:40+00:00    APP    None   \n",
       "3  2025-07-30T05:09:10+00:00  2025-07-30T05:09:10+00:00    APP    None   \n",
       "4  2025-07-30T05:20:36+00:00  2025-07-30T05:20:36+00:00    APP    None   \n",
       "\n",
       "  result.label result.score                                 result.explanation  \n",
       "0         None         None                       Ignores dietary restrictions  \n",
       "1         None         None                               Incorrect formatting  \n",
       "2         None         None  Recipe provided is a bit complicated for the u...  \n",
       "3         None         None  Asks for breakfast but a lunch or dinner is su...  \n",
       "4         None         None  Does not include unique ingredients like the u...  "
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Then get all annotations (including notes) for these spans\n",
    "annotations_df = await px_client.spans.get_span_annotations_dataframe(\n",
    "    spans_dataframe=spans,\n",
    "    project_identifier=\"UHJvamVjdDoy\",\n",
    "    exclude_annotation_names=[],  # Include everything\n",
    ")\n",
    "\n",
    "# Reset index to make the index a column\n",
    "annotations_df = annotations_df.reset_index()\n",
    "\n",
    "annotations_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>context.trace_id</th>\n",
       "      <th>result.explanation</th>\n",
       "      <th>attributes.llm.input_messages</th>\n",
       "      <th>attributes.llm.output_messages</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>80cd60cc61bd68c53f44e2986a800da9</td>\n",
       "      <td>Ignores dietary restrictions</td>\n",
       "      <td>[{'message.role': 'system', 'message.content':...</td>\n",
       "      <td>[{'message.role': 'assistant', 'message.conten...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>501cc758aec7b0a713e2fd37a83052c9</td>\n",
       "      <td>Incorrect formatting</td>\n",
       "      <td>[{'message.role': 'system', 'message.content':...</td>\n",
       "      <td>[{'message.role': 'assistant', 'message.conten...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>81a7d4e6c801603fd26e543eaa2507f7</td>\n",
       "      <td>Recipe provided is a bit complicated for the u...</td>\n",
       "      <td>[{'message.role': 'system', 'message.content':...</td>\n",
       "      <td>[{'message.role': 'assistant', 'message.conten...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8dc30dea4ca3d23a321726afc460923d</td>\n",
       "      <td>Asks for breakfast but a lunch or dinner is su...</td>\n",
       "      <td>[{'message.role': 'system', 'message.content':...</td>\n",
       "      <td>[{'message.role': 'assistant', 'message.conten...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4e2aa436bf98de2ac8632d2076ae472a</td>\n",
       "      <td>Does not include unique ingredients like the u...</td>\n",
       "      <td>[{'message.role': 'system', 'message.content':...</td>\n",
       "      <td>[{'message.role': 'assistant', 'message.conten...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   context.trace_id  \\\n",
       "0  80cd60cc61bd68c53f44e2986a800da9   \n",
       "1  501cc758aec7b0a713e2fd37a83052c9   \n",
       "2  81a7d4e6c801603fd26e543eaa2507f7   \n",
       "3  8dc30dea4ca3d23a321726afc460923d   \n",
       "4  4e2aa436bf98de2ac8632d2076ae472a   \n",
       "\n",
       "                                  result.explanation  \\\n",
       "0                       Ignores dietary restrictions   \n",
       "1                               Incorrect formatting   \n",
       "2  Recipe provided is a bit complicated for the u...   \n",
       "3  Asks for breakfast but a lunch or dinner is su...   \n",
       "4  Does not include unique ingredients like the u...   \n",
       "\n",
       "                       attributes.llm.input_messages  \\\n",
       "0  [{'message.role': 'system', 'message.content':...   \n",
       "1  [{'message.role': 'system', 'message.content':...   \n",
       "2  [{'message.role': 'system', 'message.content':...   \n",
       "3  [{'message.role': 'system', 'message.content':...   \n",
       "4  [{'message.role': 'system', 'message.content':...   \n",
       "\n",
       "                      attributes.llm.output_messages  \n",
       "0  [{'message.role': 'assistant', 'message.conten...  \n",
       "1  [{'message.role': 'assistant', 'message.conten...  \n",
       "2  [{'message.role': 'assistant', 'message.conten...  \n",
       "3  [{'message.role': 'assistant', 'message.conten...  \n",
       "4  [{'message.role': 'assistant', 'message.conten...  "
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "combined_df = pd.merge(\n",
    "    spans,\n",
    "    annotations_df,\n",
    "    left_on=\"context.span_id\",\n",
    "    right_on=\"span_id\",\n",
    "    how=\"right\",  # Keep all spans, even those without annotations\n",
    ")[\n",
    "    [\n",
    "        \"context.trace_id\",\n",
    "        \"result.explanation\",\n",
    "        \"attributes.llm.input_messages\",\n",
    "        \"attributes.llm.output_messages\",\n",
    "    ]\n",
    "]\n",
    "combined_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "raw"
    }
   },
   "source": [
    "## Step 3.2: Axial Coding & Taxonomy Definition\n",
    "\n",
    "Group your observations from open coding into broader categories or failure modes. **We'll use an LLM to make this easier!**\n",
    "\n",
    "**What the LLM will do:**\n",
    "1. **Find Patterns**: Analyze all your annotations to identify common themes\n",
    "2. **Create Categories**: Generate 4-6 systematic failure mode labels\n",
    "3. **Apply Labels**: Classify each trace using the discovered failure modes\n",
    "\n",
    "**What you'll get:**\n",
    "- **Clear Title** for each failure mode\n",
    "- **One-sentence Definition** explaining the failure\n",
    "- **1-2 Examples** from your actual bot traces\n",
    "- **Labeled dataset** with each trace classified\n",
    "\n",
    "**Example failure modes:**\n",
    "- \"Dietary Mismatch\" - Bot suggests food that violates stated dietary restrictions\n",
    "- \"Missing Steps\" - Recipe instructions are incomplete or unclear\n",
    "- \"Wrong Context\" - Bot misunderstands what the user is asking for\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\"Dietary Ignored\", \"Formatting Error\", \"Complexity Mismatch\", \"Meal Type Mismatch\", \"Ingredient Omission\", \"Skill Level Misalignment\"]\n"
     ]
    }
   ],
   "source": [
    "prompt = f\"\"\"\n",
    "You are analyzing Recipe Bot failures. Look at these examples where a user queried the bot, the bot responded, and an analyst (me) described what went wrong.\n",
    "\n",
    "EXAMPLES:\n",
    "{combined_df.to_json(orient=\"records\", lines=True)}\n",
    "\n",
    "Based on the patterns you see in the analyst's descriptions of what went wrong, create 4-6 systematic failure mode labels that would be useful for categorizing these types of issues.\n",
    "\n",
    "Each label should:\n",
    "- Be short and clear (2 words max)\n",
    "- Capture a distinct type of failure pattern\n",
    "- Be applicable to multiple traces\n",
    "\n",
    "Respond with a list of failure mode labels: [\"label1\", \"label2\", \"label3\", \"label4\", \"label5\", \"label6\"]\n",
    "\"\"\"  # noqa: E501\n",
    "\n",
    "\n",
    "client = openai.OpenAI()\n",
    "response = client.chat.completions.create(\n",
    "    model=MODEL_NAME,\n",
    "    messages=[{\"role\": \"user\", \"content\": prompt}],\n",
    "    temperature=0.3,\n",
    "    max_tokens=1000,\n",
    ")\n",
    "\n",
    "response_content = response.choices[0].message.content\n",
    "\n",
    "# result = json.loads(response_content)\n",
    "# failure_modes = result.get('failure_modes', [])\n",
    "# print(failure_modes)\n",
    "\n",
    "print(response_content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Dietary Ignored', 'Formatting Error', 'Complexity Mismatch', 'Meal Type Mismatch', 'Ingredient Omission', 'Skill Level Misalignment']\n"
     ]
    }
   ],
   "source": [
    "import ast\n",
    "\n",
    "failure_mode_labels = ast.literal_eval(response_content)\n",
    "\n",
    "print(failure_mode_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "🐌!! If running inside a notebook, patching the event loop with nest_asyncio will allow asynchronous eval submission, and is significantly faster. To patch the event loop, run `nest_asyncio.apply()`.\n",
      "llm_generate |██████████| 9/9 (100.0%) | ⏳ 00:05<00:00 |  1.75it/s\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>output</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Dietary Ignored</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Formatting Error</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Complexity Mismatch</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Meal Type Mismatch</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Ingredient Omission</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                output\n",
       "0      Dietary Ignored\n",
       "1     Formatting Error\n",
       "2  Complexity Mismatch\n",
       "3   Meal Type Mismatch\n",
       "4  Ingredient Omission"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create template for applying labels\n",
    "classification_template = PromptTemplate(f\"\"\"\n",
    "Look at this Recipe Bot interaction and the analyst's description of what went wrong.\n",
    "Apply the most appropriate failure mode label(s) from the provided options.\n",
    "\n",
    "USER QUERY: {{attributes.llm.input_messages}}\n",
    "BOT RESPONSE: {{attributes.llm.output_messages}}\n",
    "ANALYST'S ISSUE DESCRIPTION: {{result.explanation}}\n",
    "\n",
    "AVAILABLE FAILURE MODE LABELS:\n",
    "{failure_mode_labels}\n",
    "\n",
    "Based on the analyst's description of the issue, pick the failure mode that best apply to this case.\n",
    "\n",
    "Respond with just the label name\n",
    "\"\"\")\n",
    "\n",
    "# Run llm_generate for classification\n",
    "\n",
    "\n",
    "results = llm_generate(dataframe=combined_df, template=classification_template, model=phoenix_model)\n",
    "\n",
    "results.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "output\n",
       "Complexity Mismatch         2\n",
       "Meal Type Mismatch          2\n",
       "Ingredient Omission         2\n",
       "Dietary Ignored             1\n",
       "Formatting Error            1\n",
       "Skill Level Misalignment    1\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Count the occurrences of each failure mode label in the results\n",
    "label_counts = results[\"output\"].value_counts()\n",
    "label_counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Join results to combined_df on the index (axis=1), then rename 'output' to 'failure model'\n",
    "final_data = combined_df.join(results.rename(columns={\"output\": \"failure model\"}))\n",
    "final_data.head()\n",
    "\n",
    "final_data.to_csv(\"labeled_synthetic_data.csv\", index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "raw"
    }
   },
   "source": [
    "\n",
    "\n",
    "# Summary & Expected Outputs\n",
    "\n",
    "## What You'll Create\n",
    "\n",
    "**Files you'll generate:**\n",
    "- `generated_synthetic_queries.csv` - Your test dataset  \n",
    "- `labeled_synthetic_data.csv` - Your final analysis with failure mode labels\n",
    "\n",
    "## Steps to Complete\n",
    "\n",
    "1. **Run Part 1 code** - Generate test queries and upload to Phoenix\n",
    "2. **Part 2 (Phoenix UI)** - Run your prompt on queries, annotate problems with open coding  \n",
    "3. **Run Part 3 code** - Export traces, use LLM to discover patterns and create taxonomy\n",
    "\n",
    "## What Part 3 Creates\n",
    "\n",
    "The LLM analysis will automatically generate:\n",
    "- Failure mode categories discovered from your annotations\n",
    "- Systematic classification of each trace\n",
    "- Complete taxonomy with definitions and examples\n",
    "- Analysis spreadsheet with binary failure mode columns\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "recipe-bot-hw2",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
